The present invention relates in general to visible light communication (VLC) sending digital data, and, more specifically, to detection and isolation of multiple VLC sources in video camera images and optimizing signal-to-noise performance and data transmission speed by capturing subwindowed images for each VLC source.
Visible light based communication (VLC), also referred to as LiFi, is a wireless data communication technology being actively researched for automotive applications and for consumer electronics applications. Data transmission involves modulating (i.e., flashing) a light source such as a light emitting diode (LED) to encode data, and receiving the modulated light at a light sensor such as a photodiode or a camera to decode the data.
A vehicle having a VLC receiver might receive VLC signals from a fixed source (e.g., an LED traffic light) or from a mobile source (e.g., an LED signal light such as a taillight on another car). The data being shared may be related to traffic information or control, hazard warnings, navigation assistance, and many other types of data. A preferred image sensor is a “camera on a chip” comprising a two-dimensional array of pixels for capturing successive image frames taken at a rate that can distinguish the flashing of the light source. A camera with a wide field of view is desirable in order to detect and track a VLC image source, or even multiple sources simultaneously. In addition, multiple cameras can provide a mosaic of adjacent or overlapping images with limited fields of view that can be stitched together. A typical VLC transmitter uses a singular LED or an array of LEDs.
Complementary metal-oxide semiconductor (CMOS) image sensors are particularly advantageous since they provide good image quality with low power requirements, are low cost, and are often already present on a vehicle as an object detection sensor for other vehicle systems (e.g., an advanced driver assistance system such as a lane departure monitor, blind spot detector, adaptive cruise control, or automatic parking guidance). CMOS image sensors are also common on other types of devices which may be used as VLC receivers, such as smartphones.
A CMOS imager utilizes an image read-out process known as a rolling shutter, wherein the image exposure and read-out functions are conducted on a row-by-row basis (i.e., the rows of pixel are converted into a digital signal one row at a time). The use of a rolling shutter results in a temporal aliasing, wherein the image's pixel row/columns include a slight time delay that may capture artifacts in moving objects or changes in lighting levels in the scene since different rows within a single image frame will capture the same object at slightly different times. This property of the rolling shutter has been used to increase the data rate of a VLC transmission by flashing the LED source at a frequency corresponding to the exposure times of successive rows (requiring that the LED source spans a plurality of the pixel rows in the camera). The resulting image of the LED source consequently displays alternating bands of light and dark lines which encode successive bits in a serial data stream. An example is shown in U.S. application Ser. No. 15/975,033, filed May 9, 2018, entitled “Visible Light Communication System With Pixel Alignment For High Data Rate,” which is incorporated herein by reference in its entirety.
The outdoor environments for automotive applications of light-based communication pose several significant challenges. A primary challenge is detecting and reading data transmission from relatively dim LiFi sources in that also include bright ambient light sources such as the sun. Another challenge is there may be multiple LiFi devices transmitting simultaneously with varying light intensity. This results in a difficult environment for detecting dim signals or signals where the ON/OFF states are hard to distinguish. Alternatively, the exposure settings of the camera may result in blooming artifacts when imaging the LiFi transmitter during night time operations. Photodiodes would offer a large dynamic range in the analog-to-digital response, but they would not be able to distinguish data transmission from LED's of similar intensity since photodiodes do not detect the location of a light source, just the intensity. Moreover, a photodiode lacks the ability of a camera to localize a LiFi transmitter. While a camera sensor such as a CMOS-based device would allow discernment between two LED signals of similar intensity, it typically does not have sufficient dynamic range to operate in challenging day time environments with a great amount of ambient light. Additionally, there are other challenges relating to the varying intensity LiFi signals in an automotive environment interfering with one another and making it difficult to increase data transfer rates.